Many data sources can be divided into streams by attribute. Streams can be further divided into tokens. For example, a web server log is an example of a record stream that can be transposed into multiple attribute streams. Another example is a database containing a number of tables with streams of entries classified by attribute. Database warehouses and logging systems can be required to store very large volumes of data. The overall infrastructure cost for log storage and analysis is typically proportional to the total number of bytes occupied by log files. Good compression techniques can therefore reduce the overall infrastructure cost. One example of a compressor that can be used with tokenized streams is known as the bzip2 compressor. The bzip2 compressor uses a sorter known as the Burrows-Wheeler Transform.